Support vitest 5 by GuillaumeLagrange · Pull Request #86 · CodSpeedHQ/codspeed-node

GuillaumeLagrange · 2026-06-29T07:59:11Z

No description provided.

sheremet-va · 2026-06-30T09:42:07Z

+  }
+}
+
+function patchRunBenchmarks(): void {


By the way, you can do that just in a setup file without overriding the runner. This should also make it support the browser(?)

If there is any feedback, I would love to address them on vitest side; we would also like to use codspeed after Vitest 5 is out in our own repo

@sheremet-va thanks for the heads up, currently trying out this approach.

The main pain point I am running into is that we are struggling to highjack the benchmark runner when we are in simulation/analysis mode.

When in this mode, what we used to do is highjack the whole run function, to have a very simple sequence of

warmup

manually call optimizer

start valgrind instrumentation

execute code once

stop

But I'm struggling a bit to find a good way to replicate the behavior without overriding the whole runner.
I still do not have a proper api request, nor am I sure it's the way to go for the analysis mode, becuase maybe running browser mode under valgrind is not a good idea in the first place, and the main motivation for not highjacking the whole runner is actually to be as compatible as possible with the browser mode.

Could it be possible to have a way for the plugin to hook/customize the createTinybench function? Maybe by defining some sort of abstract "benchmark backend" interface that we could fullfill, that would default to a tinybench implementation. It would also make it easier to switch away from tinybench on your end at some point.

The way tinybench is instanciated/ran does not allow for much flexibility on our end, and I cannot find a good implementation without overriding the whole runner as we did before for our analysis mode.

I'd love to exchange over this over discord/slack/a call if you want, or we can keep things here on github.

Feel free to join the discord - https://chat.vitest.dev/

I will give you the ecosystem role, we can discuss it there. I just need your nickname

I've joined and pinged on via MP, my nickname on the server is the same is github

codspeed-hq · 2026-07-01T09:07:39Z

Merging this PR will regress 5 benchmarks

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 5 improved benchmarks
❌ 5 regressed benchmarks
✅ 175 untouched benchmarks
🆕 79 new benchmarks
⏩ 53 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	WallTime	`test sync baz 10`	108 ns	132 ns	-18.18%
❌	WallTime	`switch 1`	72 ns	84 ns	-14.29%
❌	WallTime	`switch 1`	72 ns	84 ns	-14.29%
❌	WallTime	`switch 2`	84 ns	96 ns	-12.5%
❌	WallTime	`test sync baz 100`	204 ns	228 ns	-10.53%
⚡	Simulation	`test_recursive_fibo_10`	1,028.4 µs	276.5 µs	×3.7
⚡	WallTime	`switch 1`	84 ns	72 ns	+16.67%
⚡	WallTime	`test_iterative_fibo_10`	108 ns	96 ns	+12.5%
⚡	WallTime	`test sync baz 10`	108 ns	96 ns	+12.5%
⚡	WallTime	`test sync baz 100`	216 ns	192 ns	+12.5%
🆕	WallTime	`iterative fibo 15`	N/A	672 ns	N/A
🆕	WallTime	`iterative fibo 20`	N/A	660 ns	N/A
🆕	WallTime	`recursive fibo 15`	N/A	21.1 µs	N/A
🆕	WallTime	`recursive fibo 20`	N/A	227.4 µs	N/A
🆕	WallTime	`iterative`	N/A	576 ns	N/A
🆕	WallTime	`recursive`	N/A	20.7 µs	N/A
🆕	WallTime	`iterative`	N/A	564 ns	N/A
🆕	WallTime	`recursive`	N/A	223.3 µs	N/A
🆕	WallTime	`fibo 10`	N/A	2.4 µs	N/A
🆕	WallTime	`fibo 15`	N/A	20.8 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing cod-2931-prepare-compatibility-with-vitest-5 (eb2042e) with main (d7b132e)}

53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

greptile-apps · 2026-07-19T18:51:01Z

Greptile Summary

This PR adds Vitest 5 support to the CodSpeed Vitest plugin. The main changes are:

A new Vitest 5 benchmark provider path.
Legacy Vitest 3/4 runner code moved under src/legacy.
Shared instrumentation and walltime result conversion helpers.
Updated Vitest dependency ranges and a Vitest 4 example project.

Confidence Score: 4/5

The new Vitest 5 path can record distinct benchmarks under one CodSpeed URI, and legacy fallback can skip instrumentation.

Multiple bench() registrations in one Vitest 5 group can share one CodSpeed benchmark identity.
Unknown Vitest version detection can route supported Vitest 3/4 runs through the v5 gating path.
The remaining changes are mostly layout, tests, examples, and shared helper refactors.

packages/vitest-plugin/src/v5/provider.ts and packages/vitest-plugin/src/vitestBackend.ts

Important Files Changed

Filename	Overview
packages/vitest-plugin/src/v5/provider.ts	Adds the Vitest 5 provider, but the current URI construction can collapse multiple registrations in one group.
packages/vitest-plugin/src/vitestBackend.ts	Adds version-specific backend selection, with a risky unknown-version fallback to the v5 path.
packages/vitest-plugin/src/instrument.ts	Adds shared analysis and walltime instrumentation helpers.
packages/vitest-plugin/src/index.ts	Delegates plugin apply and config behavior to the selected Vitest backend.
packages/vitest-plugin/src/legacy/walltime.ts	Moves the legacy walltime runner to the new legacy layout.
packages/vitest-plugin/src/globalSetup.ts	Makes setup and teardown idempotent for repeated Vitest 5 setup calls.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Vitest run] --> B{Detected Vitest major}
  B -->|3 or 4| C[Legacy backend]
  C --> D[legacy analysis or walltime runner]
  B -->|5 or newer| E[V5 backend]
  E --> F{benchmark enabled}
  F -->|no| G[default Vitest behavior]
  F -->|yes| H[v5 benchmark provider]
  H --> I{CodSpeed mode}
  I -->|analysis or simulation| J[run one measured pass]
  I -->|walltime| K[run tinybench with hooks]
  J --> L[CodSpeed benchmark result]
  K --> L

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
  A[Vitest run] --> B{Detected Vitest major}
  B -->|3 or 4| C[Legacy backend]
  C --> D[legacy analysis or walltime runner]
  B -->|5 or newer| E[V5 backend]
  E --> F{benchmark enabled}
  F -->|no| G[default Vitest behavior]
  F -->|yes| H[v5 benchmark provider]
  H --> I{CodSpeed mode}
  I -->|analysis or simulation| J[run one measured pass]
  I -->|walltime| K[run tinybench with hooks]
  J --> L[CodSpeed benchmark result]
  K --> L

Comments Outside Diff (2)

packages/vitest-plugin/src/v5/provider.ts, line 1778 (link)

Group URI Drops Benchmark Name

When a Vitest 5 test registers multiple bench(name, ...) entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/vitest-plugin/src/v5/provider.ts
Line: 1778

Comment:
**Group URI Drops Benchmark Name**

When a Vitest 5 test registers multiple `bench(name, ...)` entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

How can I resolve this? If you propose a fix, please make it concise.

packages/vitest-plugin/src/vitestBackend.ts, line 1910-1911 (link)

Unknown Version Selects V5

When Vitest resolution from process.cwd() fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no test.benchmark.enabled flag, so config() returns undefined and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/vitest-plugin/src/vitestBackend.ts
Line: 1910-1911

Comment:
**Unknown Version Selects V5**

When Vitest resolution from `process.cwd()` fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no `test.benchmark.enabled` flag, so `config()` returns `undefined` and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
packages/vitest-plugin/src/v5/provider.ts:1778
**Group URI Drops Benchmark Name**

When a Vitest 5 test registers multiple `bench(name, ...)` entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

### Issue 2 of 3
packages/vitest-plugin/src/vitestBackend.ts:1910-1911
**Unknown Version Selects V5**

When Vitest resolution from `process.cwd()` fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no `test.benchmark.enabled` flag, so `config()` returns `undefined` and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

### Issue 3 of 3
packages/vitest-plugin/src/globalSetup.ts:15
**Untracked TODO Comment**

This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the `TODO(COD-XXX): ...` form or remove it.

```suggestion
// TODO(COD-XXX): Check if this can be avoided
```

_{Reviews (1): Last reviewed commit: "feat(vitest-plugin): use the vitest 5 be..." | Re-trigger Greptile}

greptile-apps · 2026-07-19T18:51:06Z

+let setupHappened = false;
 let teardownHappened = false;

+// TODO: Check if this can be avoided


Untracked TODO Comment

This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the TODO(COD-XXX): ... form or remove it.

Suggested change

// TODO: Check if this can be avoided

// TODO(COD-XXX): Check if this can be avoided

Rule Used: Every TODO comment must reference a Linear issue: ... (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/vitest-plugin/src/globalSetup.ts Line: 15 Comment: **Untracked TODO Comment** This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the `TODO(COD-XXX): ...` form or remove it. ```suggestion // TODO(COD-XXX): Check if this can be avoided ``` **Rule Used:** Every TODO comment must reference a Linear issue: ... ([source](https://app.greptile.com/codspeed/-/custom-context?memory=65193bc9-f65b-477d-9521-104b5aac5931)) How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Vitest 5 reworked the benchmark backend: the dedicated `NodeBenchmarkRunner` and the `vitest/runners` / `vitest/suite` entrypoints are gone, benchmarks now run inside `test()` through the unified `TestRunner`, and tinybench moved to v6 (stats moved from `result.benchmark` to `task.result.latency`, `includeSamples` became `retainSamples`). Detect the installed Vitest generation and select the integration seam behind a `VitestBackend` abstraction so the rest of the plugin never inspects the version: - v3/4 keep the custom benchmark runner per instrument mode (analysis/walltime). - v5 installs instrumentation from a setup file that patches the shared `TestRunner.runBenchmarks` static. A setup file (rather than a custom `test.runner`) leaves the runner untouched for non-benchmark tests and also applies to the browser pool.

Now that the plugin's own dev dependency tracks Vitest 5, add a dedicated Vitest 4 example so the legacy (v3/4) benchmark seam keeps explicit coverage alongside the existing with-vitest-v3 example. Mirrors that example, pinning vitest ^4.1.9. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

A plugin behaves differently depending on whether CodSpeed drives the run, and each mode takes a different path through the plugin, so all three have to be run when developing or reviewing one. Write down the commands and the expected output per mode, and ignore the local `.codspeed` result directory those runs produce. Co-Authored-By: Claude <noreply@anthropic.com>

Vitest 5 replaced the top-level `bench()` declaration and its `NodeBenchmarkRunner` with a `bench` test-context fixture executed by a `benchmark.provider`: a module Vitest hands the registered functions, their options and the test that declared them, and whose results it treats as authoritative. Wire the V5 backend to register CodSpeed as that provider and add the provider itself, which runs the analysis and walltime modes off the registrations it receives. Benchmark URIs now carry the registration name after the test path, so a test declaring several benchmarks through `bench.compare()` reports one benchmark per registration. Move the package's own benches to the fixture API to cover both shapes, and read imported bindings into a local before the measured loop so the module-runner getters stay out of the samples. Gate the injection on the instrument mode: Vitest 5 clones its benchmark project after the Vite config hooks ran, so a benchmark run can no longer be detected from the incoming config, and CodSpeed only ever drives benchmark runs. Without CodSpeed the plugin injects nothing and Vitest runs the benchmarks through its own tinybench provider. Vitest 5 also runs `globalSetup` once per project, including the benchmark project it clones, so make setup and teardown idempotent instead of failing the run on the second teardown. The legacy (Vitest 3/4) runner keeps working: its Vitest lookups move behind `legacy/compat`, which now owns the types of the removed benchmark backend so the package type-checks against Vitest 5. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

Benchmark the plugin against Vitest 5 in CI the way the v3 and v4 examples do for their majors, using the `bench` fixture and `bench.compare()`. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

Benchmarks are declared through the `bench` test-context fixture on Vitest 5; keep the top-level `bench()` form as a note for Vitest 3 and 4. Drop the fallback log line from the sample output, which the plugin no longer prints. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

GuillaumeLagrange changed the title ~~Cod 2931 prepare compatibility with vitest 5~~ Support vitest 5 Jun 29, 2026

sheremet-va reviewed Jun 30, 2026

View reviewed changes

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch 5 times, most recently from 25ef927 to e04f0da Compare July 1, 2026 08:59

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch 2 times, most recently from f511dee to c3a356b Compare July 1, 2026 12:57

greptile-apps Bot reviewed Jul 19, 2026

View reviewed changes

HiDeoo mentioned this pull request Sep 3, 2026

Update to Vitest 5 withastro/starlight#4180

Draft

2 tasks

GuillaumeLagrange and others added 6 commits September 4, 2026 12:00

test(vitest-plugin): add with-vitest-v5 example

2f7a55b

Benchmark the plugin against Vitest 5 in CI the way the v3 and v4 examples do for their majors, using the `bench` fixture and `bench.compare()`. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch from eec7ec7 to eb2042e Compare September 4, 2026 16:46

Navigation Menu

Uh oh!

Support vitest 5 - #86

Support vitest 5#86
GuillaumeLagrange wants to merge 6 commits into
mainfrom
cod-2931-prepare-compatibility-with-vitest-5

GuillaumeLagrange commented Jun 29, 2026

Uh oh!

sheremet-va Jun 30, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

sheremet-va Jul 1, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

codspeed-hq Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 19, 2026 •

edited

Loading

Flowchart

Comments Outside Diff (2)

Uh oh!

greptile-apps Bot Jul 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	// TODO: Check if this can be avoided
	// TODO(COD-XXX): Check if this can be avoided

Uh oh!

Conversation

GuillaumeLagrange commented Jun 29, 2026

Uh oh!

sheremet-va Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

sheremet-va Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed-hq Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 5 benchmarks

Performance Changes

Footnotes

Uh oh!

greptile-apps Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Comments Outside Diff (2)

Uh oh!

greptile-apps Bot Jul 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codspeed-hq Bot commented Jul 1, 2026 •

edited

Loading

greptile-apps Bot commented Jul 19, 2026 •

edited

Loading